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The Intergovernmental Panel on Climate 
Change (IPCC) predicts that 

21 st -century global surface temperature 
change is likely to exceed 2°C 
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IPCC prediction comes from ensemble of 
global climate models: CMIP5 (Coupled 
Model Intercomparison Project) 


CMIP5 Model 

BCC-CSMl.l 

BCC-CSMl.l.m 

CanESM2 

CCSM4 

CESM1-BGC 

CESM1-CAM5 

CESM1-WACCM 

CMCC-CESM 

CMCC-CM 

CMCC-CMS 

CNRM-CM5 

ACCESS 1.0 

ACCESS 1.3 

CSIRO-Mk3.6.0 

FGOALS-g2 

FIO-ESM 

GFDL-CM3 

GFDL-ESM2G 

GFDL-ESM2M 

GISS-E2-H 

GISS-E2-H-CC 

GISS-E2-R 

GISS-E2-R-CC 

HadGEM2-AO 

HadGEM2-CC 

HadGEM2-ES 

INM-CM4 

IPSL-CM5A-LR 

IPSL-CM5A-MR 

IPSL-CM5B-LR 

MIROC5 

MIROC-ESM 

MPI-ESM-LR 

MPI-ESM-MR 

NorESMl-M 

NorESMl-ME 


Models are averaged together to make 
climate predictions 



21st-century temperature trend 
(RCP 8.5 multi-model ensemble mean) 
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But models can have a large spread in predictions, 
and individual models can perform 
very differently from observations 


Global surface temperature anomaly, from 35 CMIP5 models 




The traditional Multi-Model Ensemble (MME) 

Approach uses the model mean to provide an 
improved "best estimate" forecast 


Global average surface temperature change 
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The multi-model ensemble generally 
performs better than individual models 

Example: I 2 performance index (Reichler and Kim 2008) 

Calculates aggregated model errors relative to NCEP/NCAR 
reanalyses for multiple climate variables 


"Observations" 

(reanalysis) 
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Better performance: 
Less error 
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Some models perform better than others: 


Can we use knowledge of model performance 
for a better way to combine model output? 


Ensemble CMIP 

mean model 



Better performance: 


Less error 
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The "intelligent ensemble" method 

for creating multi-model ensemble projections 


Conventional method 
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Proposed “intelligent” method 
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NASA 

satellite 

observations 
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Project goal: 

determine future climate state 
using observed current climate 
and an ensemble of models 


fO^obs) 



Observed 

climate 


Future 

climate 

state 
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Previous work has explored model performance 
and ensemble-weighting metrics 


Several examples: 

• Model subsets (USGCRP 2009) 

• Performance metrics (Gleckler et al. 2008, Reichler and Kim 2008) 

• Constrained projections (Tett et al. 2013; Giorgi and Mearns 2003) 

• Weighted future trends (Boe et al. 2009) 

• Bias correction (Baker and Huang 2012) 


"The community would benefit from a larger set 
of proposed methods and metrics" (Knutti 2010) 
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New climate model performance 


metrics are tested: 

representative of energy budget processes 


Radiation budget quantities 

• Top-of-atmosphere (TOA) longwave 
(LW) and shortwave (SW) radiation 
fluxes 

• Surface LW and SW radiation fluxes 

• Surface temperature 


New process-oriented metrics 


Statistical tests 

• F-test for equal variances 

• Kolmogorov-Smirnov test for 
distribution similarity 

• Earth Mover's Distance (EMD): test 
for area of distribution overlap 

• Local Variance: test variance of first 
difference time series (Baker and 
Taylor 2015) 


8 TOA Radiation flux 

: represent interannual-timescale radiative feedbacks 

o Surface temperature 
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Model ddtd! 32 CMIP5 models http://pcmdi9.llnl.gov/ 

• 'Pre-Industrial Control' simulations (monthly mean, 100 years) 
to create metric weights 

• 'RCP 8.5' future simulations (monthly mean, 2081-2100 minus 
2011-2030 to produce 21 st -century trends) 


Observational datasets: 

NASA CERES EBAF-TOA and surface monthly global-mean (full data 

record: 03/2000 - 05/2014) 

http://ceres.larc.nasa.gov/ 

NASA GISS Surface Temperature Analysis (GISTEMP) 
http://data.giss.nasa.gov/gistemp/ 
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Step 1: Test model quality with selected metrics 


OLR all-sky variance test 
OLR all-sky K-S test 
OLR ali-sky local variance test 
OLR all-sky EMD value 
OLR cloudy-sky variance test 
OLR cloudy-sky K-5 test 
OLR cloudy-sky local variance test 
OLR cloudy-sky EMD value 
OLR clear-sky variance test 
OLR clear-sky K-5 test 
OLR clear-sky local variance test 
OLR clear-sky EMD value 
SW all-sky variance test 
SW all-sky K-S test 
SW all-sky local variance test 
SW all-sky EMD value 
SW cloudy-sky variance test 
SW cloudy-sky K-5 test 
SW cloudy-sky local variance test 
SW cloudy-sky EMD value 
SW clear-sky variance test 
SW clear-sky K-S test 
SW clear-sky local variance test 
SW clear-sky EMD value 
Surface temperature variance test 
Surface temperature K-S test 
Surface temperature local variance test 
Surface temperature EMD value 
OLR/Ts variance test 
OLR(cloudy-sky)/Ts variance test 
OLR/Ts K-S test 
OLR(cloudy-sky)/Ts K-S test 
OLR Ts regression means test 
OLR(cloudy-sky) Ts regression means test 
SW/Ts variance test 
SW(cloudy-sky)/Ts variance test 
SW/Ts K-S test 
SW(c!oudy-sky)/Ts K-S test 
SW T s regression means test 
SW(cloudy-sky) Ts regression means test 

Metric mean 
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Step 2: Using skill-subset of models, apply "perfect 
model" approach (Raisanen and Palmer 2001) 

Create set of potential "Earths" each with a continuous time series of observations 



1950 
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Step 2: Using skill-subset of models, apply "perfect 
model" approach (Raisanen and Palmer 2001) 

Create set of potential "Earths" each with a continuous time series of observations 



1975 2000 2025 2050 2075 

Year 


1950 
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• For each "perfect model" (potential Earth), the performance metrics 
are tested on one simulation (Pre-Industrial Control), then applied to a 
different simulation (RCP 8.5 future trends), linking present-day quality 
with a future state. 

• Metric values are used as model weights to create unequal-weight 
ensemble mean trends. 


Model 1 
Model 2 


Pl-Control 


Metric score: 

(tested against 
perfect model) 

0.8 



0.3 


Perfect 

model 
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• For each "perfect model" (potential Earth), the performance metrics 
are tested on one simulation (Pre-Industrial Control), then applied to a 
different simulation (RCP 8.5 future trends), linking present-day quality 
with a future state. 


• Metric values are used as model weights to create unequal-weight 
ensemble mean trends. 


Model 1 
Model 2 


Pi-Control 


Metric score: 

(tested against 
perfect model) 


RCP 8.5 
future trends 
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model 
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Model 

weight: 

0.8 Weighted 

mean 

0.3 trend 


Evaluated 
against 
"real" trend 



Metric-weighted ensemble means which have the least error compared 
with the "perfect model" are considered the best-performing metrics. 
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Reichler and Kim (2008) I 2 performance index 
is used to compare metric quality 

Metrics which perform well indicate a physical link between 
present-day model quality and reliability of projected trends 


Best-performing metrics: 


Worst-performing metrics: 


O 32 OLR(cloudy-sky)/Ts K-S test 
O 24 SW clear-sky EMD value 
V 31 OLR/Ts K-S test 


V 7 OLR cloudy-sky local variance test 
■ 17 SW cloudy-sky variance test 

▼ 15 SW all-sky local variance test 


Metric- 

weighted 

ensemble 

means 
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Better performance: 
Less error 



I 2 performance index value: mean across all 
"perfect model" iterations 
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Step 3: Using best-performing metric, create new 
"intelligent ensemble" projections 


OLR all-sky variance test 
OLR all-sky K-S test 
OLR all-sky local variance test 
OLR all-sky EMD value 
OLR cloudy-sky variance test 
OLR cloudy-sky K-S test 
OLR cloudy-sky local variance test 
OLR cloudy-sky EMD value 
OLR clear-sky variance test 
OLR clear-sky K-S test 
OLR clear-sky local variance test 
OLR clear-sky EMD value 
SW all-sky variance test 
SW all-sky K-S test 
SW all-sky local variance test 
SW all-sky EMD value 
SW cloudy-sky variance test 
SW cloudy-sky K-S test 
SW cloudy-sky local variance test 
SW cloudy-sky EMD value 
SW clear-sky variance test 
SW clear-sky K-S test 
SW clear-sky local variance test 
SW clear-sky EMD value 
Surface temperature variance test 
Surface temperature K-S test 
Surface temperature local variance test 
Surface temperature EMD value 
OLR/Ts variance test 
OLR(cloudy-sky)/Ts variance test 
, m R/Tf K-S tPTt 

I OLR(cloudv-skv)/Ts K-S test 
OLR Ts regression means test 
OLR(cloudy-sky) Ts regression means test 
SW/Ts variance test 
SW(cloudy-sky)/Ts variance test 
SW/Ts K-S test 
SW(cloudy-sky)/Ts K-S test 
SW Ts regression means test 
SW(cloudy-sky) Ts regression means test 
Metric mean 
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Use metric values 
as model weights 
to create unequal- 
weighted mean 
projections 
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Results: new 21 st -century projections (surface temperature) 




Difference between "Intelligent" and Equal-weight ensemble means (°C) 


"Intelligent" ensemble mean temperature trend (°C) 


Global-mean surface 
temperature trend: 3 °C 
(0.1 °C higher than the 
traditional equal-weight 
MME) 


The "Intelligent Ensemble" 
predicts about 10% higher 
regional surface temperature 
increases than MME 


0 


Contours are shaded only 
where the difference is 
statistically significant 
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Results: new 21 st -century projections (precipitation) 




"Intelligent" ensemble mean precipitation trend (cm/year) 


Difference between "Intelligent" and Equal-weight ensemble means (cm/year) 

14 


The "Intelligent Ensemble" 
predicts more intense 
precipitation increases in the 
tropics, especially in the 
South Pacific Convergence 
Zone (SPCZ) 


Contours are shaded only 
where the difference is 
statistically significant 
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Results: new 21 st -century projections (surface downward 

SW radiation) 



Difference between '‘Intelligent" and Equal-weight ensemble means (W/m 2 ) 



Higher surface radiation: 
less clouds 


The "Intelligent Ensemble" 
predicts 10-20% less clouds 
than MME over certain land 
areas, especially in 
midlatitude regions 


Contours are shaded only 
where the difference is 
statistically significant 
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Results: new 21 st -century projections (regional-mean weights) 



Difference between "Intelligent" and Equal-weight ensemble means (°C) 

0 

- 0.1 
- 0.2 
-0.3 



Regional-mean weights can 
give very different 
predictions: the US-mean 
best-performing metric 
predicts less intense 
warming than the MME 


(0.2 °C less than MME) 


Stippling indicates where the 
difference is statistically 
significant 
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Conclusions 


This project demonstrates: 

• New climate model performance metrics related to radiation 
processes are tested on the CMIP5 archive 

• Present-day model skill is linked to quality of future projections 

The results are: 

• New "intelligent ensemble" projections are created and compared 
with traditional MME projections 

• For global-mean metrics, "intelligent ensemble" projections of 
large-scale patterns remain similar, but intensity of predicted 
surface temperature, precipitation, and surface radiation increase is 

10-20% higher than the MME 

• Regional-mean metrics can produce very different projections: the 
US-mean projected warming is 3.9 °C (0.2 °C less than MME) 



